29 Jun 2019 

Book: Learning in a Digital World pp 59-81 

Editor: Paloma DiazAndri loannouKaushal Kumar BhagatJ. Michael Spector 
Singapore: Springer. 


Chapter # - will be assigend by editors 


SUPPORTING LEARNING IN EDUCATIONAL 
GAMES: PROMISES AND CHALLENGES 


AUTHOR(s) — Valerie Shute, Seyedahmad Rahimi, & Xi Lu 


Abstract: Digital, educational games have many promises (e.g., increasing students’ 
content knowledge as well as competencies like problem solving, spatial skills, 
and persistence). However, there are challenges to overcome before using these 
games more broadly in educational settings. One challenge involves identifying 
effective, theoretically-based learning supports that do not reduce the 
fun/engagement inherent in gameplay. In our chapter, we focus on the design, 
development, and testing of various types of embedded learning supports (e.g., 
animations, worked examples, formulas, interactive definitions, and videos). 
We contextualize this discussion in terms of our ongoing research with the 
game Physics Playground, currently being tested with middle- and high school 
students. We additionally elaborate on some of the challenges that we, as well 
as other educators and educational game designers, have faced in the design of 
optimal supports, concluding with ideas for future research. 
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1. INTRODUCTION 


, 


“Play is our brain’s favorite way of learning things.” —Diane Ackerman 

Can playing digital games enhance learning? This rather general question has 
been investigated in many research projects over the past couple of decades 
using various games that support different competencies, such as visual- 
spatial abilities and attention (Green & Bavelier, 2007, 2012; Shute, Ventura, 
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& Ke, 2015), persistence (Ventura, Shute, & Zhao, 2012), creativity (Kim & 
Shute, 2015), and civic engagement (Ferguson & Garza, 2011). Also, many 
research studies have used digital games to enhance students’ knowledge 
about particular concepts like physics (Shute, Ventura, & Kim, 2013), 
mathematics (Barab, Thomas, Dodge, Carteaux, & Tuzun, 2005; Ke, 2008), 
and ecosystem science (Kamarainen et al., 2013). Most of the research on the 
effectiveness of digital games to support learning have shown positive results 
(Clark, Tanner-Smith, & Killingsworth, 2016; Gee, 2003; Ke, 2013). For 
instance, Clark and colleagues (2016) found, overall, a medium effect size in 
a meta-analysis comparing the use of digital games and nongame conditions 
relative to their effects on learning. The effectiveness of the games in 
supporting learning, however, depends on certain features of games. 


As Shute and Ke (2012) pointed out, well-designed games include the 
following features: (1) ongoing interactive problem solving; (2) specific goals 
or rules which help the player focus and stay motivated to play; (3) adaptive 
challenges which keep the level of difficulty of the game in and around the 
outer boundaries of players’ ability—as the player gains new skills and 
becomes more capable, the game’s challenges become more difficult; (4) 
control by the player of game play, the game environment, and/or the learning 
experience; (5) ongoing and timely feedback; (6) uncertainty, which makes 
the game interesting, entertaining, and unpredictable; and (7) sensory stimuli 
which refer to a system of various media, e.g., graphics, sound, and animation, 
as well as a possible storyline which can keep the player on edge and 
immersed in gameplay. Because games differ in terms of their quality, not all 
games can enhance learning, thus our focus in this chapter is on the effects of 
well-designed games as learning environments (Ke, 2016; Wouters & van 
Oostendorp, 2013) or as vehicles that can be used to enhance learning (Gee, 
2003). 


Well-designed games also benefit from particular learning theories (Gee, 
2008). For example, when playing digital games, players are actively involved 
in solving specific problems (sometimes in collaboration with other players). 
In such cases, we can see the common elements of constructivist learning, 
collaborative learning, and situated learning theories (Bruffee, 1993; Lave & 
Wenger, 1991; Leemkuil & de Hoog, 2005). Moreover, the incentive systems 
embedded in the games (i.e., the reward and penalty system in the games; the 
collection of coins, trophies, badges) are supported by basic behaviorist 
learning theories (Skinner, 1978). Learning theories support how learning 
occurs in well-designed digital games. 
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Another important factor that makes digital games potentially valuable 
learning tools has to do with how popular they are among the people around 
the world regardless of age, gender, and ethnicity. For example, 97% of 
children and adolescents in the United States play a type of digital game for 
at least one hour per day (Granic, Lobel, & Engels, 2014), and 42% of 
Americans play video games regularly, or at least three hours per week 
(Entertainment Software Association, 2016). Why are digital games so 
popular? The short answer is that they are fun and often immersive (Prensky, 
2001)—either played alone or with others. Specifically, playing well-designed 
games can lead us to a state in which we lose track of time, and experience 
strong positive feelings when solving difficult problems (“aha” moments). 
This state is called flow, introduced by Csikszentmihalyi (1990). 


Learning scientists, instructional designers, and educators have reported the 
potential of digital games as learning tools to support various content 
knowledge (e.g., physics, mathematics, ecosystems) and _ various 
competencies (e.g., problem-solving skills, critical thinking, computational 
thinking, and creativity). As a result, new fields are emerging, such as game- 
based learning, game-based assessment, serious games, and educational 
games. However, the promises of digital games for learning can fall into the 
trap of “chocolate-covered broccoli” (Laurel, 2001). That is, digital games 
with poor integration of learning materials and supports can detract from the 
fun, disrupt the state of flow, and turn the game into just more instructional 
software. This issue of optimally integrating learning supports into 
educational games has caught the attention of researchers and game designers. 
Research on best practices of incorporating learning supports in digital 
games—without interrupting flow while maximizing learning—and research 
on the effectiveness of these learning supports can shed some light on answers 
to the how, what, when, and how much of providing learning supports in 
digital games. 


The purpose of this chapter is to (a) define the most common types of supports 
and their effectiveness relative to learning, (b) present an example of our own 
work implementing a learning-supports system in an educational game, (c) 
discuss how we handled various challenges that we faced when incorporation 
learning supports in our game, and (d) suggest future research that can help 
pave the way for more successful educational games. In the following sections 
of this chapter, we will elaborate on each of these topics in order. 
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2. COMMOM LEARNING SUPPORTS IN 
EDUCATIONAL GAMES 


In this section we first look at the literature to see whether learning supports 
in educational games were effective or not. Then we elaborate eight common 
learning supports used in educational games. 


2.1 Are Learning Supports in Games Effective in their 
Support of Learning? 


As discussed in the Introduction, and based on a couple of decades of research 
in game-based learning, educational games are generally viewed as effective 
learning tools (e.g., de Castell, & Jenson, 2003; Gee, 2003, Prensky, 2001). 
But what, specifically, is the effect of including explicit learning supports in 
these games? Wouters and van Oostendorp (2013) define such supports in 
educational games as comprising multiple methods and techniques that help 
to develop learners’ cognitive activities during gameplay. 


The literature on learning supports in learning environments in general is 
somewhat conflicted. Some researchers (e.g., Black & Deci, 2000) note that 
learning environments that allow for full autonomy (i.e., student control), 
without explicit supports, can be more engaging and effective environments 
than those without such freedom. Also, Clark, Tanner-Smith, and 
Killingsworth (2016) concluded from their meta-analysis that extra instruction 
(after gameplay, in the form of learning support) did not produce any 
significant learning differences between game and non-game conditions 
where compared. 


More specifically, regarding educational games, other researchers (e.g., 
Wouters & van Oostendorp, 2013) have concluded that to keep novice players 
engaged with the game, it must include learning supports. That is, digital 
games are complex and challenging environments that demand a lot of 
cognitive effort, so without any supports, learners will likely get stuck, 
frustrated, disengaged, and thus stop playing (Wouters, van Nimwegen, van 
Oostendorp, & van Der Spek, 2013). In that case, learning outcomes may be 
in jeopardy. Therefore, including supports in educational games increases the 
odds of improving learning. However, integrating supports into educational 
games is not easy, especially if we want them to not disrupt flow. 


Regarding the effectiveness of various learning supports in educational 
games, Wouters and van Oostendorp (2013) conducted a meta-analysis on the 
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topic. They selected 29 studies (with 3,675 participants) and computed 107 
pairwise comparisons to investigate the effectiveness of learning supports in 
educational games. They found a positive and moderately-weighted effect 
size of d = .34 (z = 7.26, p < .001) which suggests that the use of learning 
supports in games can, in fact, improve learning. Furthermore, Wouters and 
van Oostendorp identified 24 different types of learning supports and grouped 
them into ten categories. We briefly discuss eight of the more common types 
of support used in educational games. 


2.2 Common Types of Learning Supports Used in 
Educational Games 


There are multiple kinds of learning supports that have been used and tested 
in educational games and other kinds of learning environments. Here we 
describe a set of eight different supports that are most commonly used in 
educational games (Wouters & van Oostendorp, 2013): reflection, modeling, 
advice, collaboration, interactivity, narrative elements, feedback, and 
modality. Wouters and van Oostendorp included two other categories: 
personalization (e.g., personalized messages), and other (e.g., goal direction, 
background information, and cues). We chose not to include these two 
categories in our chapter because of two reasons. First, what we present in our 
own work relates to the eight categories listed above, and these two categories 
seem to be less used in educational games. Second, we believe that these two 
categories can be addressed in the other eight categories. For example, 
feedback and cues can be personalized. 


The first type of support is reflection. This group of supports aims to stimulate 
learners’ thinking about their performance and learning in the game. Research 
has shown that knowledge retention is improved if students are required to 
reflect on what they learned (e.g., Leemkuil, 2006). Some of the learning 
supports in games categorized under reflection include: (1) self-explanation 
(asking learners to explain to themselves—verbally or written—as they study 
a lesson/concept; Johnson & Mayer, 2010), (2) elaboration (extra task-related 
cognitive activities; Shebilske, Goett, Corrington, & Day, 1999), and (3) 
assignments (e.g., queries to find relationships between two or more variables; 
Leemkuil, 2006). These types of support can be implemented in various forms 
during gameplay (e.g., reflective questions, extra cognitive tasks, reviewing 
and discussing their answers/solutions). The point is that this group of 
supports help learners pause for a moment, analyze their answers/solutions, 
and use organizational and integrational cognitive processes to learn the 
underlying concepts within the game. 
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The second type of support is modeling. This group of supports provides an 
explication or illustration of how to solve a problem or perform a task in the 
game. The two most common supports categorized under the modeling 
category are: (1) scaffolding (Barzilai & Blau, 2014), and (2) worked 
examples (or expert solutions; Lang & O’Neil, 2008). Modeling can be 
provided either inside or outside of the game, by a peer, expert, or the game 
itself; and it can be delivered verbally, graphically, or via animated form. One 
possible criticism regarding the inclusion of worked examples in a game is 
that learners can see a solution and then replicate it without actually thinking 
about the underlying concepts being used to solve the problem. However, with 
a good reward/penalty system in place, negative effects of using worked 
examples can be minimized. Also, providing partially worked examples can 
reduce the potential negative effect of fully worked examples. This is 
described in more detail in Section 3 where we present an example of 
integrating such worked examples in our game called Physics Playground. 


The third type of support is advice (e.g., Leutner, 1993), intended to guide the 
learner in the right direction without revealing the solution. All types of advice 
(contextualized, adaptive or not) that are game-generated can be grouped 
under this category. For example, a hint can provide the learner with 
suggestions about what to do next in the game, or provide an elaborated 
explanation about possible consequences of his/her action. Advice can consist 
of a short message asking the player to focus on a particular aspect of the task, 
or give a cue about where to start. 


The fourth support category is collaboration (van der Meij, Albers, & 
Leemkuil, 2011), which may involve other players discussing the game or a 
particular level. Collaboration can help novice players figure out ambiguities 
in the game and better understand the knowledge and skills they need to learn. 
Many games allow for live chat and exchange of information among players. 
Alternatively, collaborative gameplay may be done with learners playing the 
game in dyads or small groups, then they can get involved in after-game 
discussions in online forums or in physical environments (e.g., a classroom). 


The fifth learning support type is interactivity. This category is more focused 
on giving choices and control to the learners. Any type of learning support 
which is responsive to learners’ actions can be categorized under this group. 
For example, Moreno and Mayer (2005) designed their agent-based 
multimedia game with interactivity where students, for example, had to select 
roots, stems, and leaves that best helped plants survive on the planet. Another 
group of students used a different version of the game (i.e., with no 
interactivity). They interacted with a pedagogical agent who simply showed 
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them pertinent information regarding the plants. The authors found that 
interactivity helped students learn and retain knowledge more than non- 
interactivity. 


Narrative elements comprise the sixth type of learning support, where content 
can be integrated into the storyline of a game via narratives that contain 
surprises, foreshadowing, and fantasies. The narrative of a game provides a 
cognitive framework for the learners with which they can better learn and 
remember the underlying concepts in the game (e.g., Adams, Mayer, 
MacNamara, Koenig, & Wainess, 2012). This type of support can be seen, as 
Prensky (2001) pointed out, in genres such as adventure games or role-playing 
games. 


The seventh type of learning support — and likely the most powerful one — is 
feedback, especially formative feedback which is essential for learning (Shute, 
2008). Given the high degree of interactivity existing in most games, feedback 
becomes critically important. As Shute (2008) notes, there are many types of 
feedback, but the two most common types used in educational games are 
corrective feedback (e.g., showing if an answer/solution is correct or not), and 
explanatory feedback (e.g., describing why the answer/solution was right or 
wrong). Cameron and Dwyer (2005) found statistically significant differences 
on all learning outcomes when feedback was included in the game versus 
when it was not. 


Finally, the eighth support category is modality (Ginns, 2005; Moreno & 
Mayer, 2002; Ritterfeld, Shen, Wang, Nocera, & Wong, 2009). That is, 
learning supports can be provided via different modalities (i.e., auditory, 
visual, textual) and each type can positively or negatively affect learning. For 
example, Moreno and Mayer (2002) found that learners remembered more of 
the materials, achieved better transfer, and rated more favorably virtual reality 
environments that used speech rather than on-screen text to deliver learning 
materials. Also, Ritterfeld and colleagues (2009) point out that multimodality 
is one of the most important aspects of educational game success—providing 
learners with materials via different channels. Results of their study showed 
that multimodality positively affects knowledge gains for both short-term (at 
the posttest) and long-term (follow-up test) outcomes. 


The foregoing learning supports can be personalized and adaptive to learners. 
That is, the what, the where, the how, and the when of learning supports can 
be tailored to the current needs of the learners as well as preferences. After 
conducting a moderator analysis, Wouters and van Oostendorp (2013), found 
out that among the 29 studies they examined, reflection, modeling, 
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collaboration, modality, and feedback enhanced learning, but advice, 
interactivity, and narrative did not. This does not mean that the non-significant 
learning supports types will never be useful; rather, the effectiveness of all 
learning supports is likely dependent on appropriately integrate learning 
supports into educational games. In the next section, we present an example 
of designing, developing, and implementing learning supports in a specific 
educational game. 


3; LEARNING SUPPORTS IN PHYSICS 
PLAYGROUND 


As mentioned earlier, different types of learning supports tend to promote 
learning across educational games (Wouters & van Oostendorp, 2013). 
However, details about particular features and their associated effectiveness 
of different types of learning supports are lacking in the literature (Johnson, 
Bailey, & Van Buskirk, 2017; Ke, 2016). Ke and Shute (2015) pointed out 
that next generation of educational games will likely embody two related 
functions: (1) game-based stealth assessment, and (2) adaptive learning 
supports, which are based on the results of the in-game assessment. 
Effectively integrating the assessment and associated supports must rely on an 
iterative game design process. 


In this section, we describe some of our processes related to developing, 
implementing, and testing various learning supports in the game Physics 
Playground (Shute & Ventura, 2013). 


3.1 Original Version of Physics Playground 


Physics Playground (PP) is a homemade 2D physics game designed to 
enhance qualitative physics understanding. In the original version of PP, we 
used stealth assessment technology (Shute, 2011) to measure player’s 
conceptual understanding of physics related to: (1) Newton’s laws of force 
and motion, (2) potential and kinetic energy, and (3) conservation of angular 
momentum (Shute, Ventura, Kim, & Wang, 2014). 


The nonlinear version of PP had only one game type—the sketching interface. 
The sketching levels require players to draw simple machines (i.e., lever, 
ramp, pendulum, and springboard) to guide a green ball to hit a red balloon— 
the goal in all levels. Players can win a silver or gold trophy for solving a level, 
but no trophies for failures. Crafting optimal solutions get them a gold trophy. 
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In the Chocolate Factory level (see Figure 1), players who solve it within two 
steps get a gold trophy (i.e., drag a pin to the tree branch (Step 1), draw a ramp 
from the pin following the path of the dotted line (Step 2), then the ball will 
travel along the ramp and hit the balloon). 


Fig. 1 Chocolate Factory level in Physics Playground 


Over the past decade, we have conducted various empirical studies testing the 
effectiveness of PP on a range of competencies including physics 
understanding and other competencies, such as creativity and persistence. We 
consistently found that (1) PP can foster motivation and learning, and (2) the 
embedded stealth assessment measures are reliable and valid—significantly 
correlated with external measures (see Shute et al., 2015). The goal, however, 
was to enhance the game by including targeted in-game learning supports. 
This led to new funding (NSF and IES) to design, develop, and test both 
cognitive and affective stealth-assessment-based adaptive learning supports 
(with our focus in this chapter on the cognitive supports). Over the course of 
past two years, we conducted several usability studies to design a new version 
of PP. In the following sections, we first discuss the challenges we faced and 
decisions we made along the way. Then, we will elaborate on the current 
version of PP. 


4. CHALLENGES WE FACED, AND DECISIONS 
WE MADE 


Well-designed games and good instructional design should go hand-in-hand 
(Hirumi et al., 2010; Shute, Rieber, & Van Eck, 2011). But introducing 
learning supports in a game poses two main challenges: (1) providing 
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appropriate support without giving away the answers (e.g., Hirumi et al., 
2010), and (2) ensuring alignment between learning supports and game 
mechanics (i.e., game rules) without disrupting the flow (Ke & Shute, 2015), 
particularly since the effectiveness of the supports vary depending on the 
degree of cognitive load and game flow (Ke, 2016). 


This section focuses on the specific hurdles we encountered and our decisions 
to surmount them during the development of the cognitive supports that align 
with game mechanics in PP. We describe how we sought the sweet spot 
between the land of theory (learning supports) and the land of data (results of 
several usability studies). 


4.1 Early Version of Learning Supports 


We adopted the physics competency model to undergird the systematic design 
iterations of the supports in PP. The early version of the cognitive supports 
included five different types of support (Figure 2): (1) Game tutorial, (2) 
Worked examples, (3) Hewitt videos, (4) Physics facts, and (5) Advice. 


Fig. 2 First version of cognitive support in Physics Playground 


Game tutorials resided in two separate playgrounds. The sketching tutorial 
playground consisted of six interactive tutorials (i.e., game mechanics, nudge, 
ramp, lever, pendulum, and springboard). The manipulation tutorials 
introduced essential game tools relevant to our new task type we developed 
(i.e., blower and puffer, general sliders, specific sliders, and bounciness). 


In addition to the tutorials, students could access other supports in a level via 
the “support kit” tab located at the left-hand side of screen. Clicking on the 
tab opened the support menu (Figure 3). This allowed students to access 
physics facts, worked examples, and Hewitt videos if they were in a level 
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playing for less than 5 minutes. The advice icon only appeared when the game 
detected a student was in the same level > 5 minutes. 


Fig. 3 Support menu in Flower Power level in Physics Playground 


Clicking on the Physics facts support (i.e., the dictionary icon) led to a non- 
interactive list showing all the relevant terms, definitions, and short examples. 
Clicking on the Worked Example support (i.e., the jigsaw puzzle icon) 
directed students to the solution video. Clicking on the Hewitt video support 
allowed students to watch a physics video explaining the primary concept 
related to the level. And clicking on Advice (i.e., the light bulb icon) triggered 
a short, general hint for solving a level (e.g., “Remember that a larger force 
will cause an object to accelerate faster”). 


4.2 Usability Study 1 


To examine the effects of the five cognitive supports and our new task type 
(e.g., manipulation levels), we conducted the first usability study at our 
laboratory school, Florida State University School (FSUS) at the end of the 
first year of the project. FSUS is located in Tallahassee, Florida, in an 
urban/suburban setting. It is a K-12 school whose heterogeneous student 
population represents Florida’s population demographics (50% white, 29% 
African-American, 12% Hispanic, 5% Multicultural, 3% Asian, and 0.2% 
Native American). In FSUS, 21% of middle school students and 11% of high 
school students are enrolled in the free and reduced lunch program. 
Recruitment occurred via science teachers in their classes, and flyers at the 
school. 


In the 3-day study, we observed and interviewed 24 9" to 11" grade students, 
who were either paired or played individually for a total of 150 minutes. On 
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day 3, the students completed an 18-item physics test (developed by our 
physics experts as well as our measurement experts). All gameplay and test 
data were captured in log files. We developed a think-aloud protocol detailing 
the researcher-initiated prompts on the supports, game features, new tasks and 
levels, and test items. We also recorded students’ additional comments on the 
game and technical glitches that occurred during gameplay. Such data 
triangulation allowed for a deep look at what really worked and what did not 
and gave direction on the next design phase. 


We hypothesized that the five supports would be somewhat effective in 
developing physics understanding (as measured by the physics test). However, 
the study yielded mixed results—i.e., game tutorials were viewed as generally 
helpful, and the new manipulation task types were well-received. However, 
while students clearly favored the worked examples and Hewitt videos, they 
had mixed (mostly negative) feelings toward the Physics facts and Advice. 
The data showed that while the worked examples were the most frequently 
accessed support, the other supports were rarely used. This led us to redesign 
the learning supports based on five main decisions. 


e Redesign the Support Kit Tab: None of the students opened the tab 
voluntarily—we decided to revise the color and position of the tab to make 
it clear and visually appealing. 


e Revise the Tutorials: While most students reported the tutorials were 
straightforward and clear, some had a hard time creating optimal simple 
machine(s) per level. Consequently, we created and inserted agent- 
specific tutorials in the support kit tab to remind students to review each 
when needed. 


e Redesign Physics Facts: Not surprisingly, the majority of students noted 
that the Physics facts support was boring. We decided to change the static 
definitions to a matching game for the terms. In short, they now, 
interactively, construct their definitions of terms, like a Cloze task 
(Taylor, 1953). 


e Design Reward System: Moreover, a number of students mentioned that 
they would watch the Hewitt Videos, etc. if incentives were provided. 
This motivated our design of a reward system for the game. 


e Remove Advice: Students felt that the Advice support was neither specific 
nor helpful. We decided to remove Advice and design more level-specific 
physics hints. 
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After several rounds of discussion and revision, we further refined our 
supports and came up with the second version of learning supports as shown 
in Figure 4. The new supports are highlighted in red. 


Physics Support 


Interactive 
Definitions 


oe oo 


Fig. 4 Second version of supports in Physics Playground 


In the second version, we made the following changes: 


New help system: We regrouped the supports into physics-related and 
game-related categories. We converted the Physics Facts support to a 
simpler Glossary. And as mentioned, we added animations, interactive 
definitions, and formula options to provide additional support for the 
growth of formal physics knowledge. We also replaced the support kit tab 
with a simpler Help button. Thus, the new support system provides three 
types of help: “Show me the Physics,” “Show me a Solution or Hint,” and 
“Show me Game Tips” (see Figure 13). 


Dashboard: We created a dashboard (Figure 5)—accessible from the 
main menu in the game— and called it “My Backpack.” My Backpack 
displays the player’s progress regarding estimates of current physics 
knowledge, number of levels completed and remaining, money earned, 
and a store offering customizable items (i.e., changing ball type and color, 
changing music, and changing the background image). 
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Fig. 5 Dashboard in Physics Playground (My Backpack) 


e Reward system: Research shows that game incentive structures and level 
progression are core aspects of game rule design (Ke, 2016). The game 
allows students to earn gold/silver coins when they solve a level or access 
the supports/game tutorials. The back of both coins shows the head of Sir 
Isaac Newton. One gold coin = $20, while one silver coin = $10. We 
employed dollars ($) as the game currency for familiarity. The coins 
earned will be automatically converted to dollars and appear in the money 
bag located on the dashboard. 


We conducted the second usability study to test the effectiveness of the second 
version of our learning supports. 


4.3. Usability Study 2 


In the second usability study, we observed the gameplay of 44 8" grade 
students at the same school in Usability Study 1 across three days, with a 
posttest and a questionnaire on day four. The students were assigned to two 
groups: learning support and non-learning support. Both groups played about 
40 minutes each day. 


Despite some technical issues, the results showed that students were quite 
excited and engaged when playing the game. They did note that the tutorials 
were too long and not interactive, which echoed the comments obtained from 
the first usability study. Also, like the first usability study, the learning support 
most accessed by this group accessed was “Show me a Solution” (i.e., worked 
examples). Again, the other supports were not often used. This reinforced the 
need for a good reward system operational in the game—to limit the abuse of 
worked examples, and to direct more attention to the other supports intended 
to engender physics understanding. 
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These results motivated us to make the following decisions: (a) revise and 
operationalize the reward system with a reasonable incentive scheme intended 
to increase students’ motivation to view various physics supports (e.g., we 
raised the price of a worked example from $30 to $60, changing the cost of a 
background image from $5 to $20, changing music from $15 to $40, and 
changing ball color from $30 to $60), (b) add a free hint to the “Show me a 
Solution” tab, and (c) create interactive tutorials for both sketching and 
manipulation levels (see Section 4). The current supports are shown in Figure 
6. 


| | 
F 7 Worked pee Interactive . 
Dashboard | Tutorials Examples Animations Definitions Formulas | Hewitt Videos | Glossary 


Fig. 6 Third version of supports in Physics Playground 


4.4 Usability Study 3 


Before conducting the third usability study, after looking at what we found 
from the first two studies, we developed a set of new learning supports, and a 
new set of test items (i.e., near-transfer items). The purpose of usability study 
3 was to (1) investigate the effectiveness of the new learning supports 
accessible via the Help button (i.e., seven animations explaining the energy 
can transfer [ECT] and properties of torque [POT] concepts with narrations; 
see Figure 7) when combined with game play, and (2) pilot-test our near 
transfer test items we developed (Figure 8). For these purposes, we selected 
the two minimally overlapping concepts in our competency model: ECT and 
POT. We also developed a new set of tutorials for nudge, lever, ramp, 
pendulum, and springboard. In total, students had 35 levels to complete. 
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Fig. 7 One of the new learning supports (see the video here). 


To evaluate students’ physics understanding, we used two physics test forms 
(Form A = 14 items; Form B = 14 items), each of which included 10 near- 
transfer test items (i.e., less technical, and more similar to the PP levels), and 
4 far-transfer test items (i.e., similar to the Force Concept Inventory test items; 
see Figure 9). Also, to evaluate students’ game satisfaction and learning 
supports satisfaction, we used a 16-item, Likert-scale questionnaire. 


In which picture (A or B) will the blue box bounce higher? 


a) A 

b) B 

c) Both will reach the same height. 
d) Not enough information 


Fig. 8 An example of our POT near-transfer test items. The answer is B. 
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If the lever is balanced in the picture above, which of the 
following would cause the lever to go unbalanced? 


a) Replace 4 kg with 8 kg and move it to point A 
b) Replace 4 kg with 8 kg and move it to point B 
c) Both 

d) Neither 


Fig.9 An example of our POT far-transfer test items. The answer is B. 


Our convenience sample included 14 students (6 seventh graders, 8 eighth 
graders; 6 female, 8 male) from a school of arts and sciences at Florida who 
were compensated with a $10 gift card upon the completion of the study. 
Students first completed a demographic questionnaire followed by the pretest 
in about 20 minutes. Then, all the students played the game for 75 minutes in 
two stages: (1) the first 20 minutes: getting familiar with the game through the 
tutorials and freely accessing all the learning supports, and (2) the next 45 
minutes: playing the game with accessing only the “physics supports” (in this 
stage the researchers prompted the students to access the “physics supports” 
after playing 3 levels or every 8 minutes). At the end of the gameplay, students 
completed the posttest, and the game and learning supports satisfaction 
questionnaire (all the tests were administered online using Qualtrics). 


Results showed a Cronbach’s @ of .61 for our ECT and .38 for our POT near- 
transfer items (both pre and posttest items included; the problematic items 
have been identified and revised for future use). Students scored significantly 
higher on the posttest compared to the pretest (Mpre = 0.57, Mpost = 0.63, t (13) 
= -2.20, p < 0.05, Cohen’s d = 0.60), suggesting learning occurred. Also, the 
near-transfer pretest significantly correlated with the near-transfer posttest (r 
= 0.53, p < 0.05), suggesting reliability. 


Finally, the analysis of students’ overall game and learning supports 
satisfaction showed that students really enjoyed playing the game (M = 4.24, 
SD = 0.62, where 1 = strongly disagree and 5 = strongly agree), and they saw 
the learning supports as useful and easy to use (M = 3.99, SD = 0.51). 
Moreover, males and females equally enjoyed the game. These findings have 
convinced us that we are on the right path. We plan to conduct a more rigorous 
study in the near future to examine the effectiveness of our new supports, and 
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ultimately select the supports that are most effective. Next, we describe the 
current version of the game. 


3. CURRENT VERSION OF PP 


As explained in Section 4, over the past two years, we have been designing 
and testing the effectiveness of a variety of learning supports in PP to foster 
deep, more formal understanding of Newtonian physics. We are finalizing the 
cognitive supports and working towards developing an adaptive stealth 
assessment-based level selection algorithm. To get to the current version of 
the game that was used in our usability studies, we started by establishing a 
new, broader physics competency model, compared to the sparse model used 
in the past. 


5.1 New Competency Model 


Using the Next Generation Science Standards (NGSS) as our guidepost, we 
worked with our two physics experts to select primary physics competencies 
and sub-competencies to be assessed in the new version of PP. We also 
identified all salient game behaviors (or “indicators”) that can provide 
evidence of the proficiency status of each variable in the competency model. 
After many revisions, we finally came up with the competency model shown 
in Figure 10. The model involves four primary competencies: force and 
motion, linear momentum, energy, and torque. The model serves as the 
foundation for subsequent design phases (e.g., designing and developing a 
new task type). 


Physics <|- > 


Fig.10 Competency model for Physics Playground 
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5.2 New Task-Type and Levels 


Given this expanded competency model, we needed task types that could elicit 
evidence of the new physics concepts. This resulted in the design of our new 
manipulation task type, with drawing functionality disabled. Manipulation 
tasks require players to adjust three sliders (i.e., gravity, mass, and air 
resistance), a bounciness option, and add external forces as needed (1.e., static 
and dynamic blowers, as well as puffers) to solve a level. For instance, solving 
the Frog level (see Figure 11) requires players to adjust air resistance and 
enable the bounciness function. 


Fig. 11 Frog Level in Physics Playground 
5.3 Specific Learning Supports 


Across the past two years, we developed 8 different learning supports for the 
game: (1) worked examples, (2) animations, (3) interactive definitions, (4) 
formulas, (5) Hewitt videos, (6) glossary, (7) hints, (8) new physics supports 
(as we called the new learning supports in Section 3), and (9) interactive 
tutorials. 


In line with Wouters and van Oostendorp’s (2013) categorization summarized 
in Section 2, our worked examples (i.e., short videos showing expert solutions 
per level) relate to Modeling; our hints relate to Advice; and our animations, 
formulas, Hewitt videos, and glossary relate to Modality in that each physics 
concept in the game can be presented across multiple representations of the 
targeted physics knowledge. We selected Modeling, Modality, and Hints as 


20 Chapter # - will be assigend by editors 


the main types of support to include in the game because Modeling and 
Modality appear to be the most effective supports to elevate student learning 
relative to other learning supports (Wouters & van Oostendorp, 2013). 


To access the supports while playing a level, students click the help button 
(see left panel of Figure 12) in the lower-right corner of the screen (note: 
currently accessing supports is controlled by the player but in upcoming 
studies, we will examine the effects of player- vs. game-control of the 
supports). This triggers a pop-up window showing three options: “Show me 
the Physics,” “Show me a Solution or a Hint,” and “Show me Game Tips” 
(see right panel of Figure 12). “Show me the Physics” comprises the main 
learning support — where students can learn about physics phenomena via 
multiple representations (i.e., physics animations with narration, interactive 
definitions, formulas, Hewitt videos, and a glossary). “Show me a Solution or 
a Hint” and “Show me Game Tips” focus on game-related support — where 
students can access tutorials, view reminders about game mechanics, and learn 
about “My Backpack,” the latter depicting their current progress and allowing 
them to customize the game environment. 


Fig. 12 “Help” button and help menu after the “Help” button is clicked 


Show me the Physics leads the student to the physics page showing the 
following options: “Animation,” “Definition,” “Formula,” “Hewitt video,” 
and “Glossary” (see Figure 13; note that the formula is not present if the 
concept doesn’t have an associated formula or equation). 
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Eldstic potentidl energy 


oan ae 


Animation Definition Formula Hewitt Video Glossary 


Fig. 13 “Show me the Physics” menu 


e =Physics animations. The new physics animations, with narration, connect 
the physics concepts with how they are applied in the game to solve a level 
(see Figure 7 for an example). These videos follow the same structure: (1) 
introduce the concept that will be presented in the video (e.g., “Here you 
are going to see how energy is transferred to a ball using a pendulum’), 
(2) state the concept (e.g., “gravitational potential energy is the energy of 
height...”), (3) demonstrate a failed attempt to solve a level in PP 
environment (i.e., the pendulum does not have enough angular height), 
and then (4) show a successful attempt to solve that level. 


e Interactive Definitions. An interactive task that allows students to drag 
and drop the choices to the right place and complete a definition of a 
physics term. In the upper left is the animation related to the term. 
Students watch the animation and drag the five phrases to the correct 
blanks within the definition. When the blanks are correctly filled, a 
congratulation message pops up and students see the complete definition 
of the term. 


e Formulas. Not all terms have associated formulas or formulas appropriate 
for the student level. Clicking on a formula card reveals the formula, along 
with a short explanation of each component/variable. 


e Hewitt Videos. Hewitt videos are an engaging series of cartoon videos 
explaining various physics concepts, developed by Paul Hewitt. The team 
edited the length of each video to make it illustrate one targeted 
competency only (we received Paul Hewitt’s permission to edit and use 
the videos). 


e Glossary. The glossary provides brief explanations of 28 physics terms. 
The terms have been selected, edited, and revised by the physics experts. 
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Each level is linked to only one physics term. However, students can 
access the glossary at any time. 


e Clicking on Show me a Solution or a Hint (Figure 13) opens a pop-up 
window. Based on feedback from two usability studies, we designed hints 
to help those who are struggling but are reluctant to watch the solutions. 
For instance, if a sketching level can only be solved by a springboard, the 
free level-specific hint will be: “Try drawing a springboard.” If a student 
elects to view a worked example, he or she will watch a worked example 
after paying $60 as a disincentive. All worked examples are 1-2-minute 
long. The worked examples are complete and can be viewed here on our 
YouTube channel. 


Finally, Show me Game Tips (Figure 14) is where students can find game rules, 
review game tutorial images, and learn about “My Backpack.” Clicking on the 
button leads to a page containing 2-3 tabs. “Controls,” “Simple Machines,” 
and “My Backpack” tabs are for sketching tasks, and “Tools” and “My 
Backpack” are for manipulation. 


Controls Simple Machines My Backpack 


Nudge 
To move right, RIGHT-click the ball; to 
move left, LEFT-click the ball. 


Draw an object 
LEFT-click and drag the cursor to draw. 


Make a pin 
LEFT-click on an object to make the pin 
for a pendulum. 


Fig. 14 “Show me Game Tips” menu. 


e “Controls” and “Simple Machines.” When a student clicks on the 
“Controls” tab, a scrollable page pops up showing game mechanics (i.e., 
nudge, draw an object, and delete an object for a sketching level). When 
a student clicks on the “Simple Machines” tab, four annotated images of 
the four simple machines (i.e., lever, pendulum, ramp, and springboard) 
show up. Each image is clickable and can be enlarged. Viewing the 
Simple Machines’ images the learners can quickly remember how the 
agents work and they don’t have to go through the full tutorials again. 
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e Tools. Clicking on “Show me Game Tips” when the player is in a 
manipulation level, provides rules for the sliders in manipulation tasks and 
a short explanation about other tools available (i.e., puffers and blowers). 


e My Backpack. In both sketching and manipulation levels, “Show me 
Game Tips” includes “My Backpack.” A screenshot from “My Backpack” 
will be shown with textboxes pointing at different parts of “My Backpack” 
explaining its function. 


e Game Tutorials. The tutorials are interactive levels with on-screen 
instructions. Sketching tutorials show how to draw simple machines. 
Manipulation tutorials show how to use the puffer/blower (that can exert 
a one-time and small force or a constant force), sliders (i.e., for mass, 
gravity, and air resistance), and bounciness function. Students can access 
them either from the playgrounds or their static images in “Show me Game 
Tips” button. 


We will make a decision, based on all the usability study results, about the 
best learning supports to include in the final version of the game. 


6. CONCLUSION AND FUTURE RESEARCH 


In this chapter we presented findings related to the effectiveness of 
educational games, specifically concerning those with embedded learning 
supports, and discussed various types of learning supports identified in the 
literature. Additionally, we illustrated how we designed, developed, and tested 
different learning supports in our educational game—Physics Playground. 
Although the literature is divided about supporting learning in various 
learning environments (especially exploratory environments), we concluded 
that having learning supports in educational games can have a positive impact 
on learning. Among the types of learning supports identified by Wouters and 
van Oostendorp (2013), reflection, modeling, collaboration, modality, and 
feedback have been found to consistently enhance learning. 


We detailed our efforts in designing and integrating learning supports in the 
game Physics Playground, and determined which supports worked best to 
foster learning. Our three usability studies yielded mixed results in response 
to this question, showing that while a large majority of students indeed 
enjoyed the game, the modeling learning supports (i.e., worked examples) 
were viewed as the most helpful compared to advice (i.e., hints) and modality 
(i.e., old animations, formulas, Hewitt videos, and glossary—the multiple 
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representations we developed per relevant physics concept). We also found 
that our new physics animations are effective and we are currently creating 
the rest of the videos for all the concepts. Moreover, we found that in the 
previous versions of the game, students were not adequately motivated to 
access the other more helpful learning supports (e.g., physics related supports), 
given the absence of an appropriate in-game reward system. Therefore, we are 
currently revising the game and supports to (a) further clarify and enhance the 
appearance and interactivity of the learning supports, (b) provide easier, more 
direct access to the supports, and (c) set up a compelling and functional reward 
system. 


Moving forward, there are a number of potential avenues for research in 
this area, such as determining the degree to which a reward system actually 
influences students’ play experience and motivation to access learning 
supports in the game. Towards that end, we are (1) optimizing the cognitive 
supports and the game reward system; (2) developing affective supports to 
complement the cognitive supports that we have developed (not focused on in 
this chapter); and (3) using stealth assessment technology to serve as the basis 
for an in-game adaptive algorithm that will select the best next level for a 
person—one that is not too difficult nor too easy and related to the targeted 
physics concept. 


This sampler of ongoing research will help us and the field figure out ways 
to optimize the design and delivery of learning supports that may be 
unobtrusively incorporated into games. The process should be iterative and 
provide research-backed evidence on: (1) the effects of different types of 
cognitive and affective supports that promote formal learning and enjoyment 
in educational games; (2) the timing and control of such supports (e.g., when 
should they be available, and who—computer or player—controls the delivery; 
and (3) the factors that mediate the influence of supports on learning and 
gameplay. 
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